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SUMMARY 


The  purpose  of  this  paper  is  to  _eft^ishso^l^  . 

theorems  for  the  solutions  ofMr^n*])  *  fljfTi1  ^ 

Recurrence  relations  of  this  kind  occur  In  various  dynamic 
programming  problems.  (  ) 
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ON  A  QUASI— LINEAR  EQUATION 
Richard  Bellman 


The  purpose  of  this  note  is  to  establish  some  limit  theorems 
for  ttt§  non-linear  recurrence  relations 

n 


(1)  x^(n+l)  ■  Max  £  alj(q)xj(n),  J  -  1 


»  2 »  •••»  n(  n  ^  0, 


under  certain  assumptions  concerning  the  initial  values  Cj  »  x^O), 
and  the  coefficient  matrices  A(q)  ■  ( a^ j ( q ) ) . 


Equations  of  this  type  occur  in  various  parts  of  the  theory 
of  dynamic  programming,  as  we  shall  indicate  below,  and  are,  in 
addition,  of  interest  in  furnishing  a  link  between  the  theory  of 
linear  and  non-linear  operations,  as  we  have  discussed  elsewhere, 

of  [1].  [a],  [}],  0]. 


§2.  The  Homogeneous  Equation 


Let  us  consider  the  equation 
n 


(1)  -  Max  alj(q)yJ,  1  -  1,  2,  ...,  n, 

where  we  impose  the  following  conditions. 


(2)  (a)  q  ■  (q^^  ,q2, . . .  ,qn)  runs  over  some  set  of  values,  S,  with 

the  property  that  the  maximum  is  attained  in  (1), 

(b)  od  >  m  >  a1J(q)  >  0,  i,J  -  1,2,. . .  ,n  for  q€S, 

(c)  for  any  q,  let  4>(q)  denote  the  characteristic  root  of 

A(q)  -  (aij(q))  of  largest  absolute  value,  the  Perron 
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root,  known  to  be  positive,  tta  assume  that  there  exists 
at  least  one  value  of  q  for  which  $(q)  assumes  Its  maxi¬ 
mum  for  q  €  S . 


We  shall  now  prove 


Theorem  1.  Under  these  conditions,  there  exists  a  unique  positive 
X  with  the  property  that  (1)  has  a  positive  solution,  y^>0,  i«l,2,...,n. 
This  solution  Is  unique  u£  to  a  multiplicative  constant,  and 


(3)  X  -  Max  i>(q). 
qCS 


Proof.  We  begin  by  showing  the  existence  of  a  positive  A  and  a 

positive  set  of  solutions  <fy. Consider  the  region  defined  by 
n  ^  ^ 

yi>0,  y^»l.  The  normalized  transformation 


(4) 


y{  -  [  Max  aij(q)yj  J  /  C 


n 


Max 

q 


Is  a  continuous  mapping  of  this  region  Into  itself.  Hence  there 
exists  a  fixed  point, ^y^J.  This  fixed  point  Is  a  solution  of 
(1),  with  A  the  denominator  In  (4).  Each  component  y^  Is  positive 


because  of  the  positivity  of  8jj(q). 


To  show  that  this  solution  is  unique  up  to  a  multiplicative 
constant,  let  o  be  another  solution  of  (1)  with  ^a>0  and 

z  a  positive  vector.  Let  «£q^.  be  the  set  of  values  for  which  the 
maximum  is  attained  in  (1),  and  the  similar  set  associated 
with  z.  Observe  that  we  may  have  different  sets  for  each  1.  We 


have  then 


•  •  •  ) 
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(5)  *  E  a1j(rH)yJ  >  E  i j ( <i ) y j ►  i  -  i,  ? 

AZ1  "  |I  alj(5>V 


n 


Let  ua  now  assume,  without  loss  of  generality  that  A<yu-.  Let 
C  be  a  positive  constant  chosen  30  that  one,  at  least,  of  the  com¬ 
ponents  yi“^z!  zero,  one  at  least  is  positive,  and  the  others 
are  non— negative.  This  can  always  be  accomplished  if  y  and  z  are 
not  proportional.  If  i  is  an  index  for  which  y^-6z^  Is  zero,  we 
have 


n 

(6)  o  -  yu.(y1-€z1)  >  Ay1-6yw.z1  >  5T~  &i j(q) (y j“€z j)  > 


Since  a^j(q)>0,  a  contradiction.  Hence  A  * and  y  and  z  are 
proportional . 

To  show  that  A  -  Pax  i>(q),  we  proceed  as  follow:',.  Let  yu.* 

q 

Max  ^)(q).  It  is  clear  that  A,  as  the  characteristic  root  of  some 
A^q),  satisfies  the  inequality  A  < j+*  .  Assume  that  actuallyyu> A  . 

Let  z  ■  ( ,z^, . . .  ,z  )  be  »  pciltive  characteristic  vector  associated 
with  yu  and  q  a  set  of  q— values  which  yield  yu.  -  i>(7)«  Then  we  ha 

n 


i  ve 


n 


(7)  /*: 


aijlqJzj  <  Max  ^  a^fq)^ 


Since  y^  1”,  positive,  we  can  find  a  positive  constant  m  su^h  that 
Zl<myi  for  1  ■  l,2,...,n.  Hence,  (7)  yields 


(8)  yU  <  m  Max 

q 


n 


*  mAyi 
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Thus  z i  <  my^  A/^u.  .  Iterating  this  we  obtain  z^  <  my^(  A/^u.  )^, 
for  arbitrary  k.  Since  A^u.  <1,  by  assumption,  this  yields 
-  0,  «  contradiction.  Hence  A  *yu. 

The  Recurrence  Relation 

Let  us  now  return  to  the  recurrence  relation  of  (1.1)  and 

prove 

Theorem  2.  If,  in  addition  to  the  conditions  of  (2.2),  we  assume 
that  there  is  a  unique  a  for  which  the  maximum  value  of  $>(q)  is 
attained  and  that  <^>0,  then 

(1)  x^nj^ay^”, 

as  n  — ►  od ,  where  a  Is  a  constant  dependent  upon  the  initial  values 

Proof .  Let  us  take  c^>0  without  loss  of  generality.  There  are 
then  two  positive  constants  k  and  K  such  that  ky1<c1<Ky1,i=l,2, . . . ,n 
Let  us  show  inductively  that 

(2)  kyt  An  <  x^n)  <  Kyt  An. 

Assume  that  we  have  the  result  for  n,  then 

(3)  x1(n+l)  <  KAn  Max  a^qjyj  -  KAn+1yi 

>  ^  ^ 

To  establish  the  asymptotic  behavior  we  show  that  for  n 
sufficiently  large  the  set  of  q's  which  furnish  the  maximum  in 
(1.1)  is  precisely  the  set  which  yields  A  -  Max  ^(q). 
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Assume  the  contrary.  Thl  means  that  infinitely  often  we 
employ  a  set  ^q^  v.’hich  is  not  identical  with  the  q  which  furnishes 
the  maximum  in  $>(q). 


We  then  have 
n 


(u)  xJL(n4l)  =  5^  aij(c,)xj(n) '  i:rl»2 


n 


<  <£  »!,<«>»,>  K*\ 


»n 


For 

(5) 


some  index  i  we  must  have 

£  au(5)yj 4 


n  _ 

with  strict  inequality.  For  if  >  a^^qjyj^  for  a11  the 

characteristic  root  of  A(q)  «  (a^^q))  of  largest  absolute  value, 

^(q-),  would  at  least  equal  A  -  Max  i(q),  which  would  contradict 

d 

the  assumption  concerning  the  uniqueness  of  the  maximum  of  $(q). 


Hence,  for  some  component,  say  the  first,  we  have 

(6)  x1(n+l)  <  wAn+1y1#  0  <  *  <  1. 

Since  a^^q)  >  0  for  i,J,  where  q  is  the  value  of  q  for  which 
A  •  ^(q),  we  see  that,  for  i  «  1,2,. ..,n, 

(7)  Xjln+2)  <  kA^1  ®lj(5)yi  *  ealj(J)y1  ] 


where  £  <  1. 
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If  therefore  a  set  of  a's  distinct  from  q  are  used  R  times, 
we  obtain 

(8)  x, (n)  <  ©,R  icAny. , 

J.  —  -L  X 

for  n  sufficiently  large.  Since  0  <  <  1,  if  R  is  too  large  we 

eventually  contradict  the  lower  bound  for  x^n). 

Hence  for  n  >  =  n^c^),  we  have 

(9)  x(n+l)  =  A(q)  x(n) , 

whence  the  a.ymptotlc  statement  of  (1)  follows. 

.  A  Dynamic  Programming  Problem 

Suppose  that  we  are  engaged  in  a  multi-stage  decision  process 
of  the  following  type.  At  each  stage  we  have  our  choice  of  variou 
operations,  which  we  number  i«l,2,...,K.  The  i—  operation  has  a 
probability  distribution  attached  with  the  following  properties: 

(1)  a.  There  is  a  probability  p^^  that  we  receive  k  units  and 

the  process  continues,  k-l,2,...,R, 
b.  There  is  a  probability  p^c  that  we  receive  nothing  and 
the  process  terminates. 

How  do  we  proceed  so  as  to  maximize  the  probability  that  we 
receive  at  least  n  units  before  the  process  terminates? 

Let  us  define  the  sequence 

(2)  u(n)  »  the  probability  of  attaining  at  least  n  units  before 

the  termination  of  the  process  using  an  optimal  pro¬ 


cedure  . 
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Then  using  the  Intuitive  "principle  of  optimality",  cf.  £l]  ,  [?]  , 

[j] ,  £V) ,  we  see  that  u(n)  satisfies  the  recurrence  relation 

R 

(3)  u(n)  *  Max  L  Plku(n-k)  n  >  0 
-  1,  n  <  0. 

Using  methods  similar  to  those  above,  we  see  that  for  large  n, 

( )  u  ( n )  ^  c  p  n  , 

where  p  is  the  root  of  largest  absolute  value,  necessarily  positive,  of 

(5) 

for  the  value  of  i  which  maximizes  £> . 

§5’  An  Analogue  of  a  Result  of  Markoff 


Markoff  showed  that  if 


(1)  x1(n+l) 


1 1 

£ 


auxj(n) 


n 


1 


:<1(0)>0,  with  the  conditions 

( ^  )  3^  j^O ,  5  C  ^  1  ^  "*  ^•*<-**,*>^* 


then 

O)  lim  x.  (n)  *  c,  i  -  l,2,...,n, 
n  — ►  od 

where  c  depends  on  the  initial  values. 
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The  same  proof  shows  that  the  same  result  holds  for  the 
sequence  defined  by 


(*) 


x1(u+l) 


n 


Ma 

q 


aJLj(q)Xj(n), 


provided  that  the  conditions  In  (2)  hold  uniformly  In  q.  The 
constant  will,  of  course,  In  general,  be  different  from  that  above. 
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